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Abstract 


The purpose of this project evaluation was to assess the impact of data-based individualization 
(DBI) on the mathematics achievement of students with intensive mathematics learning needs, 
including students with disabilities. The evaluation study used a cluster randomized trial in 
which elementary schools were randomly assigned to treatment using a delayed-intervention 
design. Since this was a development project, the evaluation delineated between the primary, 
confirmatory impact question and exploratory research questions. The confirmatory question 
included students in Grades 1-2 and was concerned with the relationship of one year of DBI 
implementation support in comparison with a business-as-usual, delayed intervention group. 
Because of the developmental, iterative nature of the project, exploratory questions were 
concerned with cumulative longitudinal relations between years of DBI implementation support 
between two cohorts of elementary schools. In addition, project staff supported DBI 
implementation pilot in two middle schools and tracked student progress in those sites. Analytic 
results provided preliminary evidence to suggest that there may be contextual factors that govern 
the likelihood a student will profit from DBI. In addition, schools may require significant 
ongoing support to sustain implementation. 
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Introduction 


The purpose of the Investing in Innovation and Improvement (i3) development project, Using 
Intensive Intervention to Improve Mathematics Skills of Students with Disabilities, was to 
increase the achievement and skills of students with severe and persistent mathematics learning 
needs, including students with disabilities. This project was grounded in the concept of data- 
based individualization (DBD), an iterative and systematic approach to intensive intervention that 
uses student data to determine when and how to adapt, intensify, and individualize interventions 
for “high need” students who do not respond to more standardized methods (i.e., Tier 2 
intervention) of remediation. For purposes of this project, high need students were identified by 
their lack of responsiveness to previous mathematics intervention or remediation efforts, 
disability status, or both. 


DBI is a validated approach to intensive intervention that may be implemented as a component 
of a Multi-Tiered System of Supports (MTSS).' Its origins are in data-based program 
modification and experimental teaching (Deno & Mirkin, 1977). Prior randomized controlled 
trials (RCTs) and research syntheses have shown that the DBI process is associated with 
moderate-to-large effect sizes in literacy and mathematics (0.60 to 1.10; e.g., Jung, McMaster, 
Kunkel, Shin, & Stecker, 2018; Stecker, Fuchs, & Fuchs, 2005) when implemented in controlled 
settings. In addition, two meta-analyses of mathematics interventions for students with 
disabilities and low-performing students (Baker, Gersten, & Lee, 2002; Gersten et al., 2009) 
found that providing teachers with progress monitoring data and specific feedback on students’ 
mathematics performance, which are critical components of DBI, appeared to have a positive 
impact on students’ mathematics achievement. 


The DBI process comprises five iterative steps (Peterson, Danielson, & Fuchs, 2019). Step one 
starts with the use of a validated intervention program or platform that is aligned with the 
student’s area(s) of need (e.g., mathematics calculation, reading fluency). In step two, the teacher 
monitors progress using a valid, reliable assessment tool. If the student is making adequate 
progress, the intervention continues as designed. If progress is inadequate, however, the teacher 
collects additional diagnostic data to determine the student’s instructional needs and hypothesize 
productive adaptations to the intervention program (step three). In step four, the teacher 
implements the adaptation(s), and continues to monitor progress to determine response (step 5). 
If the student is responsive, the intervention and progress monitoring continue to ensure ongoing 
progress. If response is inadequate, the teacher repeats steps 3-5, continuing to make data-based 
adaptations until the student demonstrates adequate progress. 


Despite the body of evidence supporting the efficacy of DBI in controlled studies, many schools 
struggle to implement it, particularly in mathematics. Educators often note a number of reasons 
for this difficulty, including limited time for intervention in the school day, competing priorities, 
lack of qualified staff, and availability of validated intervention and progress monitoring tools. 
Bearing these issues in mind, this project attempted to implement DBI in the area of mathematics 
while also addressing the school and district-level challenges that often impede the 
implementation of evidence-based practices. In this project, we leveraged and extended the work 
of the National Center on Intensive Intervention (NCII; www. intensiveintervention.org), leaning 


' We use the terms MTSS and Response to Intervention (RTI) interchangeably in this report. 
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on the lessons learned from previous DBI implementation efforts in schools and districts 
throughout the country. 


Formative evaluation data from both NCII and the present project have shown that school teams 
can improve their implementation of DBI when they receive ongoing support (see Petscher, 
Margolin, Kuchle, Danielson, & Zumeta Edmonds, 2017; Schumacher, Zumeta Edmonds, & 
Arden, 2017). Thus, the primary mechanism for supporting educators to increase their capacity 
to deliver DBI to improve student-level outcomes in this project occurred through extensive 
collaboration with school intervention teams, local education agency (LEA) staff, and local 
coaches. The goal of this collaboration was to develop and implement comprehensive DBI 
systems in mathematics in eight elementary schools? in our partner district, located near Tacoma, 
Washington (See Appendix A: Project Logic Model). By extension, we also anticipated that our 
implementation support could have a secondary impact on overall implementation of MTSS in 
mathematics in these schools. Four project activities supported accomplishment of this goal, 
including: 


¢ Conducting intake interviews that provided data about school readiness and baseline 
implementation of components of DBI and MTSS, and supported project planning. 


¢ Providing structured support and training to develop and increase educators’ knowledge 
and ability to implement DBI in mathematics. 


¢ Soliciting feedback from families regarding their experience with DBI and special 
education processes through a series of interviews. 


* Conducting formative and summative evaluation to plan project improvement and assess 
impact. 


In the sections that follow, we describe the various components of our project, including 
implementation activities, the evaluation impact study that occurred during our first year of 
implementation? (2015-16), and findings from related exploratory analyses and evaluation 
activities (see also Appendix B). We conclude the report with a discussion of findings, 
limitations, and suggestions for future research. 


Research Questions 


The purpose of our project evaluation was to assess the impact of DBI on the mathematics 
achievement of students with intensive mathematics learning needs, including students with 
disabilities. The evaluation study used a cluster RCT in which schools were randomly assigned 
to treatment using a delayed-intervention design. For purposes of this development grant, we 
delineated between the primary, confirmatory impact question and exploratory research 
questions. The confirmatory question included students in Grades 1—2 and was concerned with 
the relationship of one year of DBI implementation support in comparison with a business-as- 
usual, delayed intervention group. Because of the developmental, iterative nature of the project, 


? At the request of our partner LEA, and with the approval of our project officer, we also supported implementation 
activities in the district’s two middle schools from 2017 to 2019. 

3 This component of the report includes results of the NEi3 impact evaluation. The project’s required fidelity 
assessment has been provided to the NEi3 team as a separate document. 
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the exploratory questions were concerned with cumulative longitudinal relations between years 
of DBI implementation support between two cohorts of elementary schools. The first cohort 
began receiving implementation support during the 2015—16 school year, and the second cohort 
received implementation support starting in the 2016-17 school year. For purposes of this 
evaluation, students were tracked over three school years, starting in Grades 1—2 through Grades 
4—5, allowing for comparisons of students in the initial and delayed implementation cohorts. 
Mathematics achievement was the primary outcome domain, and multilevel models were used 
for analysis due to the hierarchical structure of the data, with students nested within schools. 


The primary research question for the impact study was: 


e¢ What is the effect of one year of DBI (DBI Phase I) on mathematics achievement among 
Grades 1-2 students with severe and persistent mathematics learning needs‘ in treatment 
schools in comparison with business-as-usual instruction in control schools? 


Exploratory research questions were: 


* Do additional years of school implementation (1.e., dosage) of DBI yield higher 
mathematics achievement in elementary students with intensive mathematics learning 
needs, compared with schools with fewer years of implementation? 


¢ Is there an interaction between intervention status and student characteristics (e.g., race, 
IEP status, achievement level)? 


¢ What is the pattern of achievement observed in students in middle schools implementing 
DBI?° 


Method 


Participants 


The project was conducted in a diverse, urban-adjacent public school district near Tacoma, 
Washington, within the district’s eight elementary schools and two middle schools. At the 
commencement of the project, there were approximately 28 different languages spoken in the 
district, 9% of students were English learners (ELs), 72% of students received free or reduced- 
price lunch, and 12.3% were students with disabilities. Although project staff worked with 
school intervention teams throughout the project, we were interested in the impact of this work 
on student outcomes. Thus, student-level data were collected for purposes of the impact study. 
The sample consisted of approximately 4-5 Grade 1—2 students with intensive mathematics 
learning needs® in each elementary school (n = 38) who were identified and tested over three 
school years (i.e., through Grades 4—5). In addition, we tracked a separate sample of middle 
school students (n = 44) from Grade 6 to Grade 7 (i.e., two school years). 


Students targeted to receive DBI were students with intensive mathematics learning needs, 
including those with disabilities. We considered a student to have intensive mathematics learning 


4 Including students with disabilities. 
> Middle school implementation was not part of our initial evaluation plan, so this question was added. 
® Including students with disabilities. 
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needs if he or she met one or more of the following conditions as defined by his or her teacher: 
(1) the student had an individualized education program (IEP), (2) the student demonstrated a 
lack of sufficient progress in a Tier II or other remedial intervention, and/or (3) the student was 
one of the lowest achieving students in his or her class based on school mathematics screening 
data. 


Measures 


We collected data on three measures of student mathematics achievement, including A/MSweb: 
Computation, AIMSweb: Concepts and Applications, and the mathematics computation subtest 
of the Wide Range Achievement Test-4 (WRAT). Measures were administered by experienced 

paraeducators who were trained to deliver the assessments. Assessments were scored, checked, 
and then entered into a database and checked for discrepancies to ensure accuracy. 


AlIMSweb: Computation (2010) is a general outcome measure that samples the annual curriculum 
to assess growth in mathematics calculations in Grades 1-8. Measures may be administered to 
individuals or groups and take approximately 8 minutes to complete. Construct validity ranges 
from .73—.84, and alternate form reliability ranges from .82—.90 across grades. 


AlIMSweb: Concepts and Applications (2009) is a general outcome measure that samples the 
annual curriculum to assess growth in mathematics concepts such as number concepts, geometry, 
and measurement, among others, in Grades 2-8. Measures may be administered to individuals or 
groups and take 8—10 minutes to complete. Predictive validity ranges from .60-—.80, and alternate 
form reliability ranges from .80—.88 across grades. 


The WRAT-4 (Wilkinson & Roberston, 2006) mathematics computation subtest is an 
achievement measure that comprises two parts. Part 1 is a short, individually administered 
interview intended to assess basic number and counting concepts. It is typically administered to 
students age 7 years or younger and to older students who fail to achieve a minimum score on 
Part 2 of the assessment. Part 2 includes 40 mathematics calculations problems of increasing 
difficulty. Part 2 can be group administered and takes approximately 15 minutes to complete. 
Validity for the subtest was .85—.95 against the mathematics composite of the Wechsler 
Individual Achievement Test—2 (2005), and coefficient alpha reliabilities were .87—.89 across 
parallel forms of the assessment. 


Project Activities 


Project implementation was conducted in the district’s eight elementary schools and two middle 
schools as depicted in Table 1. We summarize activities of each implementation year below. 


DBI Phase I: Initial Implementation 


During DBI Phase I, intervention teams that included building principals, special educators, 
interventionists, and school psychologists began receiving implementation training and support. 
At the beginning of the school year, i3 project staff conducted site-based needs assessments to 
determine schools’ baseline implementation of components of DBI and MTSS in mathematics 
and to determine specific support needs. Project staff used this information to plan and conduct 
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monthly professional development and coaching sessions with teams from all Cohort 1 schools. 
Initially, content of the training sessions focused on building knowledge around the steps of the 
DBI implementation process. As teams began to familiarize themselves with DBI, project staff 
began to build in specific instructional content in mathematics, including a review of intervention 
programs that could be used as instructional platforms for implementing DBI. Professional 
development sessions addressed the following topics throughout the year: 


¢ Introduction to DBI 

¢ Using academic progress monitoring data to set goals, plan, and evaluate instruction 
¢ Using diagnostic data to support instructional decision making 

¢ Mathematics intervention platforms 

e Adapting instruction to address counting and place value, basic facts, and fractions 
¢ Planning effective student intervention meetings 

¢* Reviewing student data and planning instruction 


e Site-based implementation fidelity checks 


Professional development sessions were followed up with implementation coaching sessions 
approximately once or twice monthly. These coaching sessions provided opportunities for 
intervention teams to refine processes, review data, integrate new learning, and begin student- 
level intervention planning, as appropriate. 


Waitlist Control (2015—16 only). Cohort 2 school intervention teams did not participate in 
implementation activities during the 2015—16 school year and delivered mathematics instruction 
and intervention using business-as-usual procedures. Most students participated in a core grade- 
level mathematics program, but implementation of mathematics intervention was variable. Prior 
to the project, all schools in the district had access to screening and progress monitoring 
assessments, but schools’ use of those measures also varied. 


DBI Phase IT: Ongoing Implementation’ 


During DBI Phase IJ, project staff provided ongoing implementation and professional 
development throughout the school year. The scope and sequence of professional development 
reflected needs observed during the first year of implementation. School teams received refresher 
trainings on the DBI process, use of validated intervention platforms, intensification strategies, 
and the role of diagnostic data within the DBI process. Sites also participated in site visits from 
their project coach and project staff, which included a variety of activities such as observations 
of instruction, participation in data meetings, and reviews of professional development content. 
Additional monthly professional development activities also included a combination of school 
staff presentations on their progress toward implementing DBI with students and review of select 


T Note that project activities in Implementation Years 2-4 were not a part of the impact study because Cohort 2 
received the intervention during that time period. This practice was consistent with our approved evaluation plan. 
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DBI content. Project staff used a gradual release strategy to shift ownership of the DBI process 
to school staff in preparation for moving to DBI Phase III (Follow-Up). 


DBI Phase ITI: Follow-Up 


During DBI Phase III, school teams did not receive regular monthly professional development 
and coaching from project staff. Instead, project staff focused on capacity and sustainability by 
providing coaching and implementation supports based on school team requests. These requests 
included support for data analysis at intervention meetings, assistance with the collection of 
diagnostic data, and support for integrating mathematics and behavior interventions. Cohort | 
schools also participated in an annual check of implementation fidelity in June of their follow-up 
year (2018) as a measure of sustained implementation during a time of minimal implementation 
supports. 


Middle School Implementation. At the request of district leadership, project staff provided 
training to review the i3 priorities and DBI process with the district’s MTSS team. 
Representatives across grade levels (K—12) constituting both special and general education 
attended this meeting. The objective of this review was to begin building a common language 
and process for providing intensive intervention to all students across the district. As a result, 
teams from the district’s two middle school campuses expressed interest in instituting DBI at 
their sites. Project staff discussed this proposal with their 13 project officer and determined that it 
would be a beneficial opportunity for scaling up implementation within the district. Professional 
development materials were therefore modified for a middle school audience, and 
implementation topics followed a similar sequence as that described above for the elementary 
cohorts. Middle school implementation occurred during the 2017—18 and 2018-19 school years. 
Middle schools, however, were not part of the impact study. 


Evaluation Design 


As noted, the effectiveness of the DBI implementation was evaluated in eight elementary 
schools® using a cluster RCT with a delayed-intervention design. Schools were the unit of 
randomization, which took place in August 2015. In order to optimize the randomization, the 
first step was to create four blocks of two schools each. The blocks were constructed based on 
the proportion of students eligible for free or reduced-price lunch, proportion of English 
language learners, school size, proportion of Black students, proportion of Hispanic students, and 
proportion of White students as variables for creating the blocks. Because of the known relations 
among these variables, principal component analysis was used to create a component score to 
explain the variance among the potential blocking variables. The component value is the linear 
combination from optimal weighting of the variables. The resulting component score for each 
school was used to sort schools from lowest to highest and created four blocks of two schools. 
Following these procedures, the block was used for random assignment via 
www.randomization.com to assign elementary schools within blocks to Cohort 1 or Cohort 2. 
Cohort 1 represented the immediate intervention group, while Cohort 2 represented the delayed 


8 The district’s two middle schools were added as an additional cohort in 2017-18 at the request of our partner 
district and with permission from the grant’s project officer. They were not involved in prior implementation years 
and so were not part of the impact study. 
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intervention group (waitlist control). As noted, the project implementation timeline is depicted in 
Table 1. 


The comparison in 2015—16 between DBI Phase I in Cohort 1 and No DBI in Cohort 2 was the 
key confirmatory contrast of interest and was a true experimental contrast. The remaining 
between-cohort comparisons in Years 2 and 3 were exploratory and tested how dosage related to 
differences in outcomes. This is because the four schools assigned to the control group in 2015— 
16 began receiving DBI Phase I in the second year of the project (2016-17). Thus, statistical 
analyses for Year 2 and Year 3 data were comparisons of dosage between cohorts such that 
Year 2 was a comparison of two years of DBI in Cohort 1 compared with one year of DBI in 
Cohort 2. Similarly, Year 3 was a test of follow-up in Cohort 1 compared with two years of DBI 
in Cohort 2. 


Student selection. Schools were the unit of random assignment, and to facilitate project buy in, 
we worked with school staff to identify students likely to profit from DBI for participation in the 
evaluation. Introductory training sessions were intended to allow school staff to accurately select 
students who were appropriate candidates for DBI; thus, eligible students were identified during 
December of 2015, and then tested in January 2016 (following school holidays and consent 
procedures). At the time of student selection, both Cohort 1 and Cohort 2 school staff were asked 
to use rosters from the time of randomization to select students who would be good candidates 
for DBI based on the criteria listed previously (i.e., students who had IEPs, or who had not been 
successful in a prior Tier IT intervention, or who were among the lowest performing students in 
their grade based on school mathematics screening data). No students joined the sample after 
initial identification. 


Analysis 


Primary impact. A multilevel model of students nested within schools was used to test the 
effect of one year of DBI implementation on Grades 1 and 2 outcomes. The primary impact 
model used was: 


(syij) Uy; +H; 


4 4 
Y, =Yoo + Yo (DBI ,) + > Yon (block 4 ,) + Yyo( pretest,,) + > ¥,9(student covariates 
k=2 


y 
s=2 


where Y;, is the mathematics score for student i in school j, DBIj is a dichotomous variable coded 


as 1 if school is a DBI school, and 0 otherwise, 7,9 is the estimated mean mathematics scores for 
comparison schools, 7,,is the DBI (treatment) effect, 7), represents the block effect, 7,,is pretest 


fixed effect, and v,, represents a set of selected student covariate fixed effects. Full information 


maximum likelihood was used to estimate fixed and random effects, and the Benjamini- 
Hochberg correction (Benjamini & Hochberg, 1995) was applied to all statistically significant 
effects to guard against the false discovery rate. Following the test of main effects of DBI on the 
selected outcomes, three individual tests of moderation were conducted to evaluate the extent to 
which pretest, whether the student was on an IEP, or race differences moderated the relation 
between DBI and posttest performance. Statistically significant pairwise interactions at p < .10 
were explored via simple slopes analysis to evaluate conditional differences and regions of 
significance. 
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Exploratory analyses. As a complementary approach to testing for main effects and interaction 
effects in a conditional means based multilevel model, a second set of exploratory analyses 
tested whether the impact of DBI was along levels of the conditional distribution of the 
mathematics achievement measures. The multilevel models used in the primary impact analysis 
are effective for empirically evaluating the average treatment effect; yet this approach based on 
averages may impose restrictions on interpretations. That is, most linear regression models are 
rooted in a conditional means approach that, by necessity, provides a conditional mean value of y 
(e.g., posttest) given a value of x (e.g., treatment). Although the mean is a desired property for 
estimating coefficients in a regression analysis, it is possible that associations between variables 
may vary depending on different points of the distribution of y. Quantile regression is a form of 
median regression that estimates the relations between y and x conditional on the distribution of y 
(Koenker & Bassett, 1978; Petscher & Logan, 2014). Where traditional linear regression is 
useful to answer the question, ““What is the average relation between treatment and posttest 
scores?”, quantile regression is useful to answer the question, “Does the relation between 
treatment and posttest scores vary depending on levels of the conditional posttest score?” 
Quantile regression has been applied under circumstances where a continuous outcome has been 
regressed on a dichotomous predictor. Thus, a natural extension of that model is to regress 
continuous posttest scores on a dummy-code variable of intervention effects. The estimated 
linear quantile mixed model to be used in this study was: 


4 4 
Y,. = Voor + %o1, (DBI p + Sy Voi (block (, ) Dp + Vio, ¢ pretest; ) + SS Yo, (Student covariates 


y 
k=2 s=2 

Where T represents the quantile of the conditional distribution of the dependent variable. Just as 
in the conditional means model (i.e., the primary impact), intercept and slope coefficients, as 
well as random effects, are estimated in the quantile model; however, unlike the impact analysis, 
they are not estimated conditional on the mean but rather on other points of the conditional 
posttest distribution. Similar to the primary impact models, three separate interaction effects 
were also explored in quantile regressions for pretest, IEP, and race moderation. 


(sis) Ug 5, + ije 


Exploratory analyses also tested Year 2 and Year 3 impacts of DBI in Cohort 1 compared with 
lesser dosages of DBI in Cohort 2. That is, Year 1 provided data that conformed to a formal test 
of impacts in the RCT. Data from Year 2 allowed for a comparison of mathematics scores for 
those schools with two years of implementation compared with one year. Similarly, data from 
Year 3 afforded a comparison between schools that were in follow-up with those receiving a 
second year of DBI. Years 2 and 3 comparisons used the primary impact and interaction 
multilevel models for end-of-year comparisons. 


Middle school data collected on students who began receiving DBI during project Year 3 were 
analyzed using a one-sample t-test. The absence of a comparison group necessarily precluded a 
between-group differences test; however, we opted to compare the middle school data to 
benchmark performance for moderate risk. Benchmarks for the fall, winter, and spring 
benchmarks for Grades 6 and 7 in AIMSweb: Mathematics Computation and Mathematics 
Concepts and Applications were used as the population value to which the sample mean was 
compared. 
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The Ime4 package (Bates, Maechler, Bolker, & Walker, 2015) in R software was used for all 
primary impact models, and the lqgmm package (Geraci, 2014) was used for the multilevel 
quantile models. 


Results 


Descriptive Statistics 


Group-based means and standard deviations are reported in Table 2 for each of the measures 
used in the evaluation report. During Year | of implementation, the mean pretest for students in 
the DBI condition for AIMSweb: Computation was 11.09 (SD = 11.04) compared with 21.10 (SD 
= 13.77) for students in the comparison group. The standardized difference between groups was 
g = 0.80 and indicated that random assignment did not produce baseline equivalence on this 
outcome measure. The What Works Clearinghouse (WWC) has provided guidance that baseline 
equivalence is empirically supported when absolute difference in effect sizes are <= 0.05. When 
the absolute baseline effect size value is calculated up to 0.25, one may use a covariate adjusted 
score (e.g., the primary impact model previously described); however, when the absolute effect 
size exceeds 0.25, the data do not satisfy criteria for baseline equivalence (U.S. Department of 
Education, Institute of Education Sciences, 2019). As such, one of two options presented for our 
study: (1) attempt to create a quasi-experimental analytic sample using propensity score 
matching, or (2) opt not to analyze the data. Due to the small, balanced analytic sample, 
propensity score matching was less plausible in terms of producing equivalent groups; thus, we 
opted not to use the mathematics computation outcome for evaluation purposes. 


The AIMSweb: Concepts and Applications assessment was administered to Grade 2 students. 
Students in the DBI condition had a mean score of 5.14 (SD = 4.33) compared with 4.94 (SD = 
4.07) for the comparison students. The standardized effect size of g = 0.05 met the WWC 
guidelines for baseline equivalence. Similarly, the WRAT baseline assessments met baseline 
equivalence metrics for the total score (g = -0.30), the Part 1 score (g = -0.09), and the Part 2 
score (g = -0.14). For each of these outcomes, the primary impact and exploratory models were 
estimated. 


Year 1 Impact Results 


Table 3 reports findings from the primary impact and moderation models for Grade 2 AIMSweb: 
Concepts and Applications. No statistically significant effect of DBI was observed in the impact 
model (-0.53, p = .703). Further, no significant interaction was estimated for baseline moderation 
(-0.15, p = .589), IEP moderation (0.82, p = .716), or race moderation (-1.13, p = .216). No main 
effects were observed on the WRAT total score (0.32, p = .685; Table 4), nor were there 
significant interactions; however, significant interactions were observed for baseline moderation 
on Part 1 of the WRAT score (0.32, p = .051; Table 5) and on Part 2 of the WRAT score (-0.58, 
p = .021; Table 6). A simple slopes analysis (Table 29) looked at the regions of significance at 
levels of one standard deviation above (1SD) and below (-1SD) the mean as well as at the mean 
across levels of DBI and comparison groups. Based on these pretest regions, there were no 
significant differences between DBI and comparison at -1SD (-0.55, p =.357), the mean (0.16, 

p =.740), or 1SD (0.87, p = .161) for WRAT Part 1 in Year 1. When considering the simple 
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slopes of WRAT Part 2 in Year 1 (Table 29), there were no significant differences between DBI 
and comparison at -1SD (1.07, p =.055)’ and the mean (-0.08, p = .861), but there were 
significant differences at 1SD in favor of the comparison group (-1.23, p =.033). 


The majority of DBI coefficients and interactions were not statistically significant, but the 
multilevel quantile results (Tables 7—22) yielded several important findings. The pretest 
moderation model for the WRAT total score (Table 12) showed significant interactions at the .50 
quantile (-0.73, p = .028) and the .75 quantile (-0.96, p = .010) of the WRAT, suggesting that the 
impact of pretest WRAT on the relation between DBI and posttest was stronger at higher 
conditional quantiles of the WRAT total score. Baseline moderation was also significant for 

Part 1 of the WRAT at the .25 quantile (0.46, p = .021) and the .50 quantile (0.31, p = .082), 
suggesting that the impact of moderation was stronger at lower conditionals quantiles of the 
WRAT Part 1 (Table 16). Table 18 shows a significant interaction for DBI by race at the .75 
quantile (-0.40, p < .10), and Table 20 shows a significant interaction for DBI by pretest at the 
.75 quantile (-0.54, p = .087). 


Year 2 Cohort Comparisons 


Results for Year 2 comparisons between Cohort | (1.e., two years of DBI implementation) and 
Cohort 2 (i.e., one year of DBI implementation) on AI[MSweb: Concepts and Applications are 
reported in Table 23. Results showed no significant effect in the primary impact model (2.39, p = 
.196); no significant interactions were observed as well. 


Year 3 Cohort Comparisons 


A significant interaction was estimated in the Year 3 comparison for DBI by race (-9.75, p = 
.006; Table 24) for Aims Web: Concepts and Applications. Simple slopes analysis (Table 28) 
showed that non-White students in Cohort 1 schools statistically outperformed non- White 
students in Cohort 2 schools (4.27, p = .018). Conversely, there was no difference in scores for 
White students between Cohorts (-5.48, p = .072). A significant interaction between DBI and 
baseline was observed for the WRAT total (Table 25) with results indicating that Cohort 1 
students with -1SD baseline scores did score as well as Cohort 2 students (-3.02, p = .004; Table 
28) and no differences between cohorts for those at the mean (-1.42, p = .060) or 1SD (0.18, p = 
.872). A significant interaction for baseline was estimated on the WRAT Part 1 (0.37, p < .001; 
Table 26) with simple slopes analysis showing that Cohort 1 students performed lower than 
Cohort 2 students at -1SD of baseline (-1.13, p < .001), but no differences were noted between 
cohorts at the mean (-0.34, p = .144) or ISD (0.45, p = .169). No significant interactions were 
noted for WRAT Part 2 (Table 27). 


Although the conditional values for the simple slope analysis used one standard deviation thresholds, we opted to 
do a secondary test given the result of the WRAT Part 2 hypothesis test. When using a 1.25 SD threshold value, the 
DBI students who were -1.25 SD on baseline statistically out-performed individuals who did not receive DBI (1.35, 
p =.038). As well, students who were 1.25 SD above the mean on baseline and received DBI performed lower on 
the post-test compared to those who did not receive DBI (-1.52, p = .026). 
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Middle School Findings 


Results from the one sample t-test for middle school students are reported in Table 28. The mean 
scores for the A[MSweb: Computation subtest tended to show increases from fall to winter and 
decreases from winter to spring. Across the six time points, only one test was statistically 
significant (i.e., Grade 6, winter; p = .048), suggesting that students’ mean performance was 
above the benchmark threshold. No significant differences were observed for the AIMSweb: 
Concepts and Applications assessment. 


Discussion 


Our results suggest that contextual factors may govern the likelihood that a student will be 
successful in DBI. With respect to the primary impact research question, results revealed no 
significant main effects for condition. In addition, the lack of pretreatment equivalence on the 
AIMSweb: Computation limited our ability to compare the two groups on that measure. At the 
same time, findings from exploratory analyses revealed interesting results that may warrant 
further investigation. Given the small sample and developmental nature of this project, all 
findings should be interpreted with caution and would benefit from further controlled study. 


First, findings from the Year 1 WRAT Part 2 analyses suggest a potential small positive effect of 
DBI for students performing well below the mean and a positive effect for the comparison group 
for students performing ISD above the mean. Although we did not observe a significant effect at 
1SD below the mean, the effect was reliable in favor of DBI for students performing 1.25 SD 
below. These trends may have implications for identifying students likely to benefit from DBI 
versus those who may benefit from less intensive levels of support. Given the low power and 
developmental nature of these data, we believe these findings are noteworthy and could benefit 
from replication with a larger sample. In particular, future research could investigate potential 
approaches to determining screening cut points to identify students with immediate need for 
DBI. In addition, it may be useful to determine the extent to which gated screening procedures or 
dynamic assessments such as those described by Fuchs, Fuchs, and Compton (2012) may help to 
improve timely, accurate identification of students. 


Given the differential impacts of DBI on higher and lower achieving students observed, it may 
also be useful to further investigate the extent to which interventions are appropriately intensive 
and aligned with students’ needs. Throughout the project, teachers reported lack of alignment 
between core instruction and intervention materials. We investigated this issue and confirmed 
poor alignment related to both mathematics instructional practices and vocabulary in three 
commonly used programs in our partner district. (See Appendix B and Nelson, Pfannenstiel, & 
Zumeta Edmonds, 2020, for more information.) Thus, we believe future efforts to implement 
DBI in mathematics may benefit from more intentional planning to ensure that core instruction 
and intervention efforts complement one another. 


Another finding of interest was the positive effect of the third year of DBI observed for non- 
White students on A[MSweb: Concepts and Applications (a proximal outcome measure) in 
Cohort 1. This finding suggests that there may be an additive positive effect of intervention over 
time for non-white students. Due to low statistical power, ELs were not analyzed separately in 
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this sample, so it is also possible that EL status and race were conflated. Thus, it may be useful to 
also investigate the interaction between EL status and DBI in future work. 


We also observed that lower achieving students (-1SD below the mean) in Cohort | performed at 
lower levels than Cohort 2 students on the WRAT in Year 3. Keeping in mind the quasi- 
experimental nature of these data compared with the experimental data from Year 1, it should be 
noted that Year 3 was a follow-up year for Cohort 1 schools, meaning that these schools did not 
receive regular implementation supports during that time period. Anecdotally, we also observed 
a decline in implementation fidelity in some of our Cohort 1 schools during our annual 
implementation checks that year. (See Appendix B for a description of these implementation 
checks.) In addition, turnover of several critical district leaders during this time period may have 
further undermined efforts to sustain implementation. Thus, it is possible that these results 
suggest a need for schools to receive ongoing support and clear district-level policies to help 
sustain implementation of DBI. 


Finally, our middle school findings were limited by the lack of control condition and the fact that 
we monitored students at their chronological grade levels rather than their instructional levels. 
This issue may have been particularly problematic for older students whose performance is often 
several years below grade level, making it difficult for the measures to detect change. 
Furthermore, we observed that many middle school teachers tended to assume that students had 
mastered skills that many had not, and they were often hesitant to reteach foundational skills. 
Many teachers also struggled to use student-level data to inform intensification and adaptation of 
intervention because they felt tied to delivery of their core curriculum and pacing guide, which 
made it difficult for them to adapt instruction. Future work at the middle school level should 
attempt to address these implementation challenges and should track students at their 
instructional levels to increase the likelihood of detecting progress if it occurs. 


Limitations 


Although we observed some potentially interesting findings, we believe they should be 
considered in light of several important limitations. First, the developmental nature of the project 
meant that we worked with a small number of schools and tracked a small sample of students. 
Consequently, statistical power was low. Furthermore, our partner district serves a very transient 
student population and, as a result, attrition was high throughout the project. Attrition among 
elementary students was approximately 50% over three years and was nearly 30% over two years 
for middle school students. Therefore, our findings must be interpreted with caution, and future 
attempts to replicate or extend this work should include larger samples to enhance power and 
help control for attrition. 


Furthermore, although schools were blocked and randomly assigned to conditions, our project 
fidelity data suggested that some Cohort 2 schools appeared to enter the project with higher 
baseline implementation of elements of MTSS and DBI-related processes. This may have 
impacted their uptake of the intervention in Years 2 and 3 and could be indicative of critical 
differences in leadership and management in these schools. Given that each cohort had only four 
elementary schools, these differences may have biased some findings in favor of the control 


group. 
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In addition, while our efforts to monitor implementation fidelity (see Appendix B) allowed us to 
look at systemic features of implementation, we were not able to assess the extent to which DBI 
was implemented with fidelity at the individual student level. In other words, although schools 
may have made promising changes to their systems (e.g., time allocated for mathematics 
intervention, adoption of evidence-based intervention platforms), the extent to which DBI 
procedures (e.g., delivering intervention aligned to student needs, implementing timely 
instructional changes based on data) were actually applied to specific students remains unclear. 
Given this and the variation in outcomes we observed across students, we believe future efforts 
to assess fidelity should include a more systematic evaluation of student-level implementation. 
Furthermore, future efforts to evaluate DBI should be sufficiently powered to allow measures of 
implementation fidelity to be included as a covariate in analytic models. 


Conclusion 


Findings from our project evaluation suggest that the impact of DBI may be differential and 
localized based on student characteristics such as prior achievement level and race. Our findings 
point to a need for screening procedures to identify students likely to benefit from DBI and for 
more robust procedures to ensure that individual students receive intensive interventions that are 
appropriately aligned to their needs. Furthermore, we suspect that schools may require additional 
sustained support to maintain implementation over time. Given the critical need to continue to 
improve outcomes for students with intensive learning needs, we hope this project has helped to 
lay the groundwork for productive future lines of research to address these ongoing questions. 
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Table 1. Project Implementation Timeline 


Implementation School Year | Implementation Phase 
b. 
015-16" Cohort 1”: ie ae I 
Cohort 2: Waitlist control/ No DBI 
Cohort 1: DBI Phase II 
AUAOE NE Cohort 2: DBI Phase I 


Cohort 1: DBI Phase III (Follow-Up) 
2017-18 Cohort 2: DBI Phase II 
Middle School Cohort: DBI Phase I 
Middle School Cohort: DBI Phase I 
* This was the NEi3 impact study year. 
> Cohorts 1 and 2 comprised four elementary schools, respectively. 


American Institutes for Research Using Intensive Intervention to Improve Mathematics Skills of Students With 
Disabilities: Project Evaluation Report—21 


Table 2. Descriptive Statistics by Group and Measure 


| DBI__|__ Comparison 
Contrast Measure Year Administration 


Combined Grades 1-2 Nie comely : Fall 
Computation 


Spring 
Grade 2 Mathematics CAP 


Pretest 
Posttest 


Combined Grades 1-2. | WRAT Part 1 
Combined Grades 1-2 | WRAT Part 2 


Follow-up 
Pretest 
Posttest 
Follow-up 


1 
2 
3 
Combined Grades 1-2 | WRAT Total 1 Pretest 
Posttest 
3 Follow-up 
1 
3 
1 
3 
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Table 3. Impact and Moderation Models for Grade 2 Mathematics Concepts and Applications 


Impact Baseline Moderation IEP Moderation Race Moderation 
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Table 4. Impact and Moderation Models for WRAT Total Score 


Impact Baseline Moderation IEP Moderation Race Moderation 
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Table 5. Impact and Moderation Models for WRAT Part 1 Score 


Impact Baseline Moderation IEP Moderation Race Moderation 


Pret [055 [ooo [<ooor] 033 [oas[ oon | 052 [0.10 <o0or| 053 [010 | <oo0r| 
roar | 009 [oas[ sor | 016 [oar oss | -009 [os2| osos | 042 [oss 000 | 
areca | | | ox Joafeoi[ | | || [| 
arp ff pts fos fore | 
mer Poo oe | 
OT 
ee re a a a a ee a Ce a TA 
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Table 6. Impact and Moderation Models for WRAT Part 2 Score 


Impact Baseline Moderation IEP Moderation Race Moderation 


Famer) | sas iis | <oon | soa [iis | <oom | si4 | 109[ <o001| 696 [1.67 | <0001 | 
TGrade | 055 os7[ 0331 | 057 [oss[ os [104 [ooo| os | 047 [os7| 0414 | 
rar | 002 [oa oes | -009 [oso | os79 [006 [or [ o9a4 | 129 [21 | 0289 | 
parr || | 058 foasfoom[ | [| [| | | | 
ame [|p foe | 
mere fore Poss | 
a 
Peace 30 F025 [0283 | 
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Table 7. Multilevel Quantile Impact Model for Grade 2 Mathematics Concepts and 
Applications 


95% CI 
Parameter Std. Error Upper Bound Pr(>|t|) 


Intercept 
Block 
Pretest 
DBI 
Intercept 
Block 
Pretest 
DBI 


Intercept 
Block 
Pretest 
DBI 
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Table 8. Multilevel Quantile Baseline Moderation Model for Grade 2 Mathematics 
Concepts and Applications 


| 95% CI CI 
Std. Lower Upper 
Parameter Value Error Bound Bound Pr(>|t|) 


Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 
Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 


Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 
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Table 9. Multilevel Quantile IEP Moderation Model for Grade 2 Mathematics Concepts 
and Applications 


| 5% CI CI 
Std. Lower Upper 
Parameter Value Error Bound Bound Pr(>|t|) 


Intercept 
Block 
Pretest 
DBI 

IEP 
IEP*DBI 
Intercept 
Block 
Pretest 
DBI 

IEP 
IEP*DBI 


Intercept 
Block 
Pretest 
DBI 

IEP 
ITEP*DBI 
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Table 10. Multilevel Quantile Race Moderation Model for Grade 2 Mathematics Concepts 
and Applications 


| 5% CI CI 
Std. Lower Upper 
Parameter Value Error Bound Bound Pr(>|t|) 


Intercept 
Block 
Pretest 
DBI 

Race 
Race*DBI 
Intercept 
Block 
Pretest 
DBI 

Race 
Race*DBI 


Intercept 
Block 
Pretest 
DBI 

Race 
Race*DBI 
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Table 11. Multilevel Quantile Impact Model for WRAT Total Score 


95% CI 
Parameter | Value | Std. Error Upper Bound | Pr(>|t)) 


Intercept 
Block 
Pretest 
DBI 


Intercept 
Block 
Pretest 
DBI 


0.50 Intercept 
Block 
Pretest 
DBI 
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Table 12. Multilevel Quantile Baseline Moderation Model for WRAT Total Score 


| 95% CI CI 
Lower Upper 
Parameter Value | Std. Error Bound Bound Pr(>|t|) 


Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 
Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 


Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 
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Table 13. Quantile IEP Moderation Model for WRAT Total Score 


[edt [ode [ ede 
(Intercept) 18.87" 19.84" 20.84" 
fee | tw | ass | aso 
Block -1.00° 0.00 0.00 
pe | ws | cas | 35 
Pretest 1.67" 2.00" 2.00" 
pee | em | ox | au 


DBI 1.00 0.00 1.00 
IEP 0.67 -1.00 -1.00 
wan ae eS 
DBI* IEP -1.00 -1.00 -1.00 
ae ae ee 


“» < 0.001, “p < 0.05, 
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Table 14. Quantile Race Moderation Model for WRAT Total Score 


eat [ Moa [Mead 
(Intercept) 16.86" 22.00° 20.83" 
fo | ans | om | aos 
Block -0.50 -0.89 -0.21 
Pe | as | ass | aay 
Pretest legis 2.00° DOF 
pee | oy | ee | ary 


DBI 1.00 -1.71 -0.18 
Race 0.38 -0.38 -0.14 
Pe ee 
DBI*Race -0.25 0.60 0.41 
nl ee ee 


*p < 0.001 
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Table 15. Multilevel Quantile Impact Model for WRAT Part 1 


95% CI 
Std. Lower Upper 
Quantile | Parameter Value Error Bound Bound Pr(>|t|) 


Intercept 
Block 
Pretest 
DBI 
Intercept 
Block 
Pretest 
DBI 


Intercept 
Block 
Pretest 
DBI 
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Table 16. Multilevel Quantile Baseline Moderation Model for WRAT Part 1 


| 95% Cl CI 
Std. Lower Upper 
Parameter Value Error Bound Bound Pr(>|t|) 


Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 
Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 


Intercept 
Block 
Pretest 

DBI 
Pretest*DBI 
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Table 17. Quantile IEP Moderation Model for WRAT Part 1 


waa [ideas 
(Intercept) 194° 13.09" 13.96" 
hale ae 
Block -0.20 -0.14 0.00 
P| as | as | ais 
Pretest 0.70" 0.71" 0.50° 
ee eee 


DBI -0.10 0.57 0.50 
IEP 0.90 0.71 0.50 
wae ae eS 
DBI* IEP 0.10 -0.43 -0.50 
a ee 


“p< 0.001 
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Table 18. Quantile Race Moderation Model for WRAT Part 1 


weeded 
(Intercept) 13.95" hes he 13.40°° 
foe | Tes | an | aan 
Block -0.33 0.15 0.07 
Pe | ase | con | ae 
Pretest 0.67" Ope 0.59" 
pee | oy | es | en 


DBI -1.33 1.28 1.33* 
Race -0.17 0.31 0.18* 
a 
DBI*Race 0.50 -0.41 -0.40* 
pees | cao | oo | ay 


“y < 0.001, ‘p < 0.1 
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Table 19. Multilevel Quantile Impact Model for WRAT Part 2 


95% CI 
Parameter | Value | Std. Error Upper Bound | Pr(>|t)) 


Intercept 
Block 
Pretest 
DBI 
Intercept 
Block 
Pretest 
DBI 


Intercept 
Block 
Pretest 
DBI 
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Table 20. Multilevel Quantile Baseline Moderation Model for WRAT Part 2 


| 95% CL CI 
Std. Lower Upper 
Parameter Error Bound Bound Pr(>|t|) 


Intercept 
Block 
Pretest 
DBI 
Pretest*DBI 
Intercept 
Block 
Pretest 
DBI 
Pretest*DBI 


Intercept 
Block 
Pretest 
DBI 
Pretest*DBI 
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Table 21. Quantile IEP Moderation Model for WRAT Part 2 


wea [ Moa [Mad 
(Intercept) 5.25" 5.91" 7.68" 
Be | es [am | a 
Block -0.33 -0.29 -0.11 
Pe | ay | as | sy 
Pretest 1.00° 1.14° 1:22" 
pee | os | ow | eo 


DBI 0.33 1.00 -0.33 
IEP -1.00 -0.71 -1.00 
Pee eee 
DBI* IEP -0.00 -0.86 -0.22 
ae a ee 


“p< 0.001 
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Table 22. Quantile Race Moderation Model for WRAT Part 2 


wea [ied [odes 
(Intercept) 10. 781" 7.89°° 
ae ee 
Block -0.73° -0.59° -0.33 
Pe | se | aso | 35 
Pretest 1.18°*° 135" eee 
mae Se ee 


DBI -1.24 -1.21 -1.00 
Race 20:42." -0.51° -0.33 
aa eae Se 
DBI*Race 0.15 0.56 0.67 
an eS 


“y < 0.001, “p < 0.05, ‘p < 0.1 
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Table 23. Year 2 Impact (March 2017) and Moderation Models for Grade 2 Mathematics Concepts and Applications 


ea ee ee 


Derren | | | | oa [oasfo2m[ | | || 
me [pi oe fora | 
err | | ||| |_| 3 [amos [| 
wie [0 fo 
erwine | neat 
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Table 24. Year 3 Impact (January 2018) and Moderation Models for Grade 2 Mathematics Concepts and Applications 
Impact Model Baseline Moderation IEP Moderation Race Moderation 


Pretest | 0.06 [021 fore [014 [oss | 0676 | 00s [022 [osx [oo [o20| oat | 
perme | | | | 0 foaloms| | | ~ | | | 
Se RS A (RA E51 
perme | ae fos 
0.08 | 
0.06 | 


Derr 
A 
oerwine | | | | | |. 91554 [ane 
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Table 25. Year 3 Impact (May 2018) and Moderation Models for WRAT Total 


Impact Model Baseline Moderation IEP Moderation Race Moderation 


pare | | oan lone 
Dame [|| om foo 
Fa A Od 
erwin || [oe | 
Pwnie | nrc [0286 | 
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Table 26. Year 3 Impact (May 2018) and Moderation Models for WRAT Part 1 


Impact Model Baseline Moderation IEP Moderation Race Moderation 


Fanterepe | 15.79 [0.0] <woor | 1559 [049 f<oani| 1556 [063| <ooo| 1595 [0.63 | <0.00r | 
PBiock | 0.06 [os] o@ss [0.11 [oar [0296 | -006 [o.14[ oot | 008 [ois | 0576 | 
perme | | |_| 037 jowfearl | [| || | | 
emer [|| oe foo fome | 
mer ot foe Poe | 
erwine || fos | oie | 
Pwnie | ts oes [0a | 
nce as Ot 
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Table 27. Year 3 Impact (May 2018) and Moderation Models for WRAT Part 2 


Impact Model Baseline Moderation IEP Moderation Race Moderation 


PBiock | 0.56 036] 0.120 | -049 [oar ous: | -oar [oas[ 0232 | 060 [028 015 | 
Pretest | 14 [oar |<noor [096 [oso] 0002 [091 [o22[ <ooor | 1.10 [023 | <00or | 
rGrade | 181 [oss] oasi [186 [ose [ oom [267 [ose | 0.003 | 2.00 [056 | 0.036 | 


Derren || | | 030 foarfoas[ | [| - | | 
emer [fos fr Poe 
mer | [0.058 
erwin || ||| | |i os _fio fore 
wie [PP 0s 0 Ps 
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Table 28. One-Sample ¢-Test Results for Mathematics Computation and Concepts and Applications in Grades 6 and 7 


Assessment Grade N Wave Sample Mean Benchmark t-Statistic 


pera 6 26 | Fall 
24 =| Winter 
24 =| Spring 
18 Fall 
18 | Winter 
20 ‘| Spring 
26 
24 
24 
18 
18 
20 


Mathematics 
Application 


Fall 
Winter 
Spring 
Fall 
Winter 
Spring 
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Full WRAT 
Ptl Baseline 
Moderation 


Full WRAT 
Pt2 Baseline 
Moderation 


Full WRAT 
Total 
Baseline 
Moderation 


Full WRAT 
Part 1 
Baseline 
Moderation 


American Institutes for Research 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 


Table 29. Simple Slopes Analysis for Moderation Probing 


Project 
Year | Moderation | Region Parameter | Estimate SE | z-Value 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 


Simple 
Intercept 


Simple Slope 
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Project 
Year | Moderation | Region Parameter | Estimate z-Value 


isp: joel 15.63 | 0.57 | 27.50 | <.001 
Intercept 
Simple Slope 0.45 0.33 Lot 0.169 


Simple Slope 


Non-White Pe a 
G2 
3 Mathematics Simple Slope 
Concepts and Simple 
Application Intercept 
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Appendix A 


Project Logic Model (Program: DEV96: Intensive Intervention in Mathematics)!° 


Existing RTI/MTSS 
structures in place in 
schools 


Resources: 

e Release time for 
intervention teams 
to meet and attend 
training and 
coaching sessions 
Support staff time 
for coordination 
Training materials 
Trainers 
Coaches 


Readiness & buy-in: 

e Ensure support from district and school 
leaders and buy-in from relevant staff. 

e Provide information about project goals and 
proaress updates. 


Professional development training to 

develop knowledge and skills to implement 

DBI in mathematics, including: 

e New pedagogical strategies 

e New strategies for using data to 
individualize intervention 

e Use of ongoing progress monitoring and 
diagnostic assessment data 


Formative evaluation: 

e Complete an initial needs assessment. 

e Determine schools’ progress with DBI 
implementation. 

e Identify strengths and challenges to inform 
future olannina. 


Implementation coaching: 
e Engage in planning, review of data, 


Proximal Outcomes/Mediators 


School/District: 


Improved systems for 
serving students with 
intensive mathematics 
learning needs 
Increased appropriate 
identification and 
placement of students 
across tiers of RTI 


Intervention teams: 


Data use to individualize 
and evaluate 
mathematics 
instruction/intervention 
Higher staff confidence in 
ability to address 
intensive mathematics 
learnina needs 


Families: 


Distal Outcome 


Student achievement: 

e Improved 
mathematics 
achievement for 
students with 
disabilities who have 
intensive 
mathematics 
learning needs 
Improved 
mathematics 
achievement for 
other students with 
intensive 
mathematics 
learning needs who 
do not have 
identified disabilities 


Families: 
Increased satisfaction 


with educational 
programming for 
students with 
disabilities 


Enhanced participation in 
education planning, 
programming for children 
with disabilities and with 
intensive mathematics 
learning needs 


evaluation of progress, and student-level 
planning. 

Support identification and implementation of 
school or district policies and procedures, 
as needed. 

Develop strategies for engaging families 
and involving them in intervention planning 
and progress reviews. 


‘0 Note: This logic model came from our approved NEi3 evaluation plan. 
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Appendix B 


Other Project Evaluation Activities 


Throughout the implementation of this project, we engaged in three formative evaluation 
activities that were outside the scope of our planned impact evaluation but that provided 
additional useful context for our work. We describe these activities in the sections that follow. 


Monitoring DBI and MTSS Implementation Progress 


As noted in the description of project activities, project staff conducted intake interviews to 
determine baseline performance of components of DBI and MTSS in mathematics. In addition, 
they conducted annual implementation pulse check interviews in the spring of each 
implementation year to formatively evaluate implementation fidelity progress of systemic 
features of DBI and MTSS in mathematics. 


Pulse checks took place prior to implementation to determine baseline performance and annually 
in the spring of each implementation year through interviews with school implementation teams. 
A member of the project staff and each site’s project coach led the interviews using a set of 
questions derived from implementation fidelity rubrics and interviews, which were initially 
developed by the National Center on Intensive Intervention and the Center on Response to 
Intervention.'! Interview participants included members of each school site’s intervention team 
(e.g., principals, school psychologists, intervention specialists, special educators), who had 
participated in ongoing professional development and coaching throughout the year. Following 
the interviews, interviewers independently rated sites’ implementation against rubric indicators. 
Scores ranged from 1—5, with “1” indicating little or no implementation and “5” indicating 
strong or full implementation. After scoring independently, interviewers compared their scores, 
discussed, and rectified any discrepancies. The average implementation score for components of 
DBI and MTSS, respectively (e.g., assessment, data-based decision making), were then 
calculated. Those subscores were then used to compute an overall average implementation score. 
On average, schools scored 2.4 at baseline for indicators of DBI in mathematics (range of 1.2— 
3.6) and improved to an average score of 3.7 (range of 3.0-4.5) by the end of the project. For 
indicators of MTSS in mathematics, schools started with an average baseline score of 2.8 (range 
of 1.8—-3.9) and improved to an average score of 3.7 (range of 3.0-4.5).'” 


Trends showed that as schools achieved growth on indicators of DBI (e.g., intensification 
practices, decision rules, using individualized student data to inform decisions), they also tended 
to show growth on indicators related to general MTSS processes such as having school-wide 
screening and progress monitoring assessments and procedures in place, holding regular data 
meetings, and providing time for Tier 2 intervention to occur. We observed this trend across 
nearly all elementary and middle school sites, providing evidence to suggest that implementation 


'! Modified to reflect MTSS terminology. 

2 The district had been implementing tiered supports in reading for several years prior to project commencement, 
although district leaders noted challenges with Tier 3 (i.e., intensive intervention), which was part of the impetus for 
their interest. Given this, we were not surprised by their higher initial overall MTSS implementation scores. 
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of MTSS may not need to be linear (1.e., that Tier 1 must be in place before Tier 2, and so forth). 
This observation contrasts with conventional wisdom about MTSS, which typically suggests that 
implementation must start at Tier 1. This approach may be problematic because it can delay 
implementation of more intensive tiers of support for students who need them. We believe a 
comparison of approaches to implementation warrants further study. 


Study of Alignment Between Mathematics Instruction and Intervention 
Materials 


Lack of alignment between instruction and intervention programs is a common barrier to 
implementation of interventions for struggling students. Throughout the course of this project, 
several school and district staff noted this concern in reference to their core mathematics 
program and their typical intervention programs. To better understand and document these points 
of alignment and misalignment, project staff conducted an analysis of these programs across 
grades 1-5, with a specific focus on their use of mathematics instructional practices and 
mathematics vocabulary. Findings from this review revealed that teachers’ perceptions about 
misalignment were accurate. We observed little overlap between programs both in terms of their 
use of mathematics practices, and mathematically precise vocabulary, where overlap between 
programs ranged from just 6.3% to 24%. A full discussion of this analysis, including 
instructional recommendations for addressing points of misalignment, was reported in Nelson et 
al., (2020). 


Gathering Feedback About DBI Implementation Through Interviews With 
Families 


To better understand parents’ perceptions regarding DBI implementation and their experiences 
collaborating with their children’s schools, we conducted a series of phone interviews with 
parents of students receiving intensive intervention in mathematics in participating schools. 
School principals and other staff involved in DBI implementation recommended parents for the 
interviews and provided parents’ contact information. We invited these parents to participate in 
telephone interviews regarding their perceptions and experiences communicating and 
collaborating with their children’s schools. We provided parents a one-page description of the 
interview and asked them to return a signed consent form to indicate their willingness to 
participate. Nineteen parents returned signed consent forms, and we were successful in 
conducting interviews with 15 of these parents. Interviewees had children in eight schools in the 
district; of these schools, seven were elementary schools and one was a middle school. Five of 
these parents had children who were identified as having a disability, 12 were parents of 
elementary school students, and three were parents of middle school students. We asked parents 
for their perceptions regarding what worked well when communicating with their children’s 
schools and what they perceived as barriers to successful parent-school communication and 
collaboration. Interviews lasted approximately 20 minutes. 


Overall, parents reported satisfaction with the amount of communication they received from their 
children’s schools. Practices that worked well for parents included frequent written 
communication with teachers, meeting with the intervention team during fall conferences to 
discuss intervention planning, sharing academic data and discussing the data at conferences, and 
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demonstrating commitment to learning about students’ needs and individualizing instruction. 
Across interview participants, there was significant variation in the amount of communication 
that parents reported receiving. For example, one parent reported receiving weekly written 
progress updates while another reported that progress updates were received only with report 
cards. 


Several parents discussed the positive impact that they perceived from the intensive intervention 
in mathematics that their children had received. For example, parents reported that their children 
showed more confidence in mathematics, were able to complete homework independently, were 
happier about going to school, and understood more mathematics concepts due to the 
individualized instruction they received. Several parents spoke about how their children needed 
more extensive explanation of mathematics concepts than was available during core instruction 
and were pleased that intensive intervention provided this opportunity. Parents also were 
appreciative of how interventionists took the time to get to know their children’s needs and to 
tailor instruction to meet those needs. 


The interviews also revealed several barriers and challenges to successful communication and 
collaboration with school staff. A few parents noted a lack of communication with intervention 
teachers or special education teachers. For example, one parent commented that he/she had never 
met the interventionist and had little idea about what work was being done during the 
mathematics intervention. In addition, some parents reported that communication from the 
general education teacher was not always relevant to their children who were participating in 
intervention outside of the general education classroom. Another barrier reported by a few 
parents was a perception that some school staff lacked the skill or patience to work with their 
children with intensive needs, especially relative to behavior issues. Other reported challenges 
included confusion about progress monitoring for students performing below grade level and 
frustration regarding the special education eligibility process. We provide a more detailed 
discussion of our findings and related recommendations in Weingarten, Zumeta Edmonds, and 
Arden (2020). 
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